Online target-speech extraction method based on auxiliary function for robust automatic speech recognition

ABSTRACT

Provided is a target speech signal extraction method for robust speech recognition including: receiving information on a direction of arrival of the target speech source with respect to the microphones; generating a nullformer by using the information on the direction of arrival of the target speech source to remove the target speech signal from the input signals and to estimate noise; setting a real output of the target speech source using an adaptive vector as a first channel and setting a dummy output by the nullformer as a remaining channel; setting a cost function for minimizing dependency between the real output of the target speech source and the dummy output using the nullformer by performing independent component analysis (ICA) or independent vector analysis (IVA); setting an auxiliary function to the cost function; and estimating the target speech signal by using the cost function and the auxiliary function.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a pre-processing method for targetspeech extraction in a speech recognition system or a speech recognitionapparatus, and more particularly, a target speech extraction methodcapable of reducing a calculation amount and improving performance ofspeech recognition by performing independent component analysis by usinginformation on a direction of arrival of a target speech source.

2. Description of the Prior Art

With respect to an automatic speech recognition (ASR) system, since muchnoise exists in real environments, noise robustness is very important tomaintain. In many cases, degradation in performance of recognition ofthe speech recognition system are mainly caused from a differencebetween a learning environment and the real environment.

In general, in the speech recognition system, in a pre-processing step,a clear target speech signal which is a speech signal of a targetspeaker is extracted from input signals supplied through input meanssuch as a plurality of microphones, and the speech recognition isperformed by using the extracted target speech signal. In speechrecognition systems, various types of pre-processing methods ofextracting the target speech signal from the input signals are proposed.

In a speech recognition system using independent component analysis(ICA) of the related art, outputs signals as many as the input signalsof which the number corresponds to the number of microphones areextracted, and one target speech signal is selected from the outputsignals. In this case, in order to select the one target speech signalfrom the output signals of which the number corresponds to the number ofinput signals, a process of identifying which direction each of theoutput signals are input from is required, and thus, there are problemsin that a calculation amount is overloaded and the entire performance isdegraded due to error in estimation of the input direction.

In a blind spatial subtraction array (BSSA) method of the related art,after a target speech signal output is removed, a noise power spectrumestimated by ICA using a projection-back method is subtracted. In thisBSSA method, since the target speech signal output of the ICA stillincludes noise and the estimation of the noise power spectrum cannot beperfect, there is a problem in that the performance of the speechrecognition is degraded.

On the other hand, in a semi-blind source estimation (SBSE) method ofthe related art, some preliminary information such as directioninformation is used for a source signal or a mixing environment. In thismethod, known information is applied to generation of a separatingmatrix for estimation of the target signal, so that it is possible tomore accurately separate the target speech signal. However, since thisSBSE method requires additional transformation of input mixing vectors,there are problems in that the calculation amount is increased incomparison with other methods of the related art and the output cannotbe correctly extracted in the case where preliminary informationincludes errors. On the other hand, in a real-time independent vectoranalysis (IVA) method of the related art, permutation problem acrossfrequency bins in the ICA is overcome by using a statistic modelconsidering correlation between frequencies. However, since one targetspeech signal needs to be selected from the output signals, problemsexist in the ICA or the like.

SUMMARY OF THE INVENTION

The present invention is to provide a method of accurately extracting atarget speech signal with a reduced calculation amount.

According to an aspect of the present invention, there is provided atarget speech signal extraction method of extracting the target speechsignal from the input signals input to at least two or more microphones,the target speech signal extraction method including: (a) receivinginformation on a direction of arrival of the target speech source withrespect to the microphones; (b) generating a nullformer for removing thetarget speech signal from the input signals and estimating noise byusing the information on the direction of arrival of the target speechsource; (c) setting a real output of the target speech source using anadaptive vector w(k) as a first channel and setting a dummy output bythe nullformer as a remaining channel; (d) setting a cost function forminimizing dependency between the real output of the target speechsource and the dummy output using the nullformer by performingindependent component analysis (ICA); and (e) estimating the targetspeech signal by using the cost function, thereby extracting the targetspeech signal from the input signals.

In the target speech signal extraction method according to the aboveaspect, preferably, the direction of arrival of the target speech sourceis a separation angle θ_(target) formed between a vertical line in afront direction of a microphone array and the target speech source.

In the target speech signal extraction method according to the aboveaspect, preferably, the nullformer is a “delay-subtract nullformer” andcancels out the target speech signal from the input signals input fromthe microphones.

In the target speech extraction method according to the presentinvention, in a speech recognition system, a target speech signal can beallowed to be extracted from input signals by using information of atarget speech direction of arrival which can be supplied as preliminaryinformation, and thus, the total calculation amount can be reduced incomparison with the extraction methods of the related art, so that aprocess time can be reduced.

In the target speech extraction method according to the presentinvention, a nullformer capable of removing a target speech signal frominput signals and extracting only a noise signal is generated by usinginformation of a direction of arrival of the target speech, and thenullformer is used for independent component analysis (ICA), so that thetarget speech signal can be more stably obtained in comparison with theextraction methods of the related art.

The present invention is to provide a non-transitory computer readablestorage media having program instructions that, when executed by aprocessor of a speech recognition apparatus or a speech recognitionsystem, cause the processor to perform the target speech signalextraction method according to the above-mentioned aspect of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configurational diagram illustrating a plurality ofmicrophones and a target source in order to explain a target speechextraction method for robust speech recognition according to the presentinvention.

FIG. 2 is a flowchart sequentially illustrating an algorithm accordingto a target speech extraction method for robust speech recognitionaccording to the present invention.

FIG. 3 is a table illustrating comparison of calculation amountsrequired for processing one data frame between a method according to thepresent invention and a real-time FD ICA method of the related art.

FIG. 4 is a configurational diagram illustrating a simulationenvironment configured in order to compare performance between themethod according to the present invention and methods of the relatedart.

FIGS. 5A to 5I are graphs illustrating results of simulation of themethod according to the present invention (referred to as ‘DC ICA’), afirst method of the related art (referred to as ‘SBSE’), a second methodof the related art (referred to as ‘BSSA’, and a third method of therelated art (referred to as ‘RT IVA’) while adjusting the number ofinterference speech sources under the simulation environment of FIG. 4.

FIGS. 6A to 6I are graphs of results of simulation the method accordingto the present invention (referred to as ‘DC ICA’), the first method ofthe related art (referred to as ‘SBSE’), a second method of the relatedart (referred to as ‘BSSA’), and a third method of the related art(referred to as ‘RT IVA’) by using various types of noise samples underthe simulation environment of FIG. 4.

FIGS. 7A and 7B illustrate a subband clique and a harmonic clique as twotypical clique cases.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to a target speech signal extractionmethod for robust speech recognition and a speech recognitionpre-processing system employing the aforementioned target speech signalextraction method, and independent component analysis is performed inthe assumption that a target speaker direction is known, so that a totalcalculation amount of speech recognition can be reduced and fastconvergence can be performed.

Hereinafter, a pre-processing method for robust speech recognitionaccording to an exemplary embodiment of the present invention will bedescribed in detail with reference to the attached drawings.

The present invention relates to a pre-processing method of a speechrecognition system for extracting a target speech signal of a targetspeech source that is a target speaker from input signals input to atleast two or more microphones. The method includes receiving informationon a direction of arrival of the target speech source with respect tothe microphones; generating a nullformer by using the information on thedirection of arrival of the target speech source to remove the targetspeech signal from the input signals and to estimate noise; setting areal output of the target speech source using an adaptive vector w(k) asa first channel and setting a dummy output by the nullformer as aremaining channel; setting a cost function for minimizing dependencybetween the real output of the target speech source and the dummy outputusing the nullformer by performing independent component analysis (ICA);and estimating the target speech signal by using the cost function,thereby extracting the target speech signal from the input signals.

In a target speech signal extraction method according to the exemplaryembodiment of the present invention, a target speaker direction isreceived as preliminary information, and a target speech signal that isa speech signal of a target speaker is extracted from signals input to aplurality of (M) microphones by using the preliminary information.

FIG. 1 is a configurational diagram illustrating a plurality ofmicrophones and a target source in order to explain a target speechextraction method for robust speech recognition according to the presentinvention. Referring to FIG. 1, set are a plurality of the microphonesMic.1, Mic.2, . . . , Mic.m, and Mic.M and a target speech source thatis a target speaker. A target speaker direction that is a direction ofarrival of the target speech source is set as a separation angleθ_(target) between a vertical line in the front direction of amicrophone array and the target speech source.

In FIG. 1, an input signal of an m-th microphone can be expressed byMathematical Formula 1.

$\begin{matrix}{{{X_{m}\left( {k,\tau} \right)} = {{\left\lbrack {A(k)} \right\rbrack_{m\; 1}{S_{1}\left( {k,\tau} \right)}} + {\sum\limits_{n = 2}^{N}\;{\left\lbrack (k) \right\rbrack_{mn}{S_{n}\left( {k,\tau} \right)}}}}},} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Herein, k denotes a frequency bin number and τ denotes a frame number.S₁(k, τ) denotes a time-frequency segment of a target speech signalconstituting the first channel, and S_(n)(k, τ) denotes a time-frequencysegment of remaining signals excluding the target speech signal, thatis, noise estimation signals. A(k) denotes a mixing matrix in a k-thfrequency bin.

In a speech recognition system, the target speech source is usuallylocated near the microphones, and acoustic paths between the speaker andthe microphones have moderate reverberation components, which means thatdirect-path components are dominant. If the acoustic paths areapproximated by the direct paths and relative signal attenuation amongthe microphones is negligible assuming proximity of the microphoneswithout any obstacle, a ratio of target speech source components in apair of microphone signals can be obtained by using Mathematical Formula2.

$\begin{matrix}{\frac{\left\lbrack {A(k)} \right\rbrack_{m\; 1}{S_{1}\left( {k,\tau} \right)}}{\left\lbrack {A(k)} \right\rbrack_{m^{\prime}1}{S_{1}\left( {k,\tau} \right)}} \approx {\exp\left\{ {j\;\omega_{k}\frac{{d\left( {m - m^{\prime}} \right)}\mspace{14mu}\sin\mspace{14mu}\theta_{target}}{c}} \right\}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 2} \right\rbrack\end{matrix}$

Herein, θ_(target) denotes the direction of arrival (DOA) of the targetspeech source. Therefore, a “delay-and-subtract nullformer” that is anullformer for canceling out the target speech signal from the first andm-th microphones can be expressed by Mathematical Formula 3.

$\begin{matrix}{{{Y_{m}\left( {k,\tau} \right)} = {{X_{m}\left( {k,\tau} \right)} - {\exp\left\{ {j\;\omega_{k}\frac{{d\left( {m - 1} \right)}\sin\mspace{14mu}\theta_{target}}{c}} \right\}{X_{1}\left( {k,\tau} \right)}}}},{m = 2},\cdots\;,{M.}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 3} \right\rbrack\end{matrix}$

The nullformer obtains the relative ratio of target speech signals byusing the information on the direction of arrival of the target speechsource, multiplies the relative ratio and an input signal of amicrophone and subtracts the multiplied value from input signals of apair of microphones so that the nullformer is configured for cancelingout the target speech source component from the input signal of amicrophone.

In order to derive a learning rule, the nullformer outputs are regardedas dummy outputs, and the real target speech output is expressed byMathematical Formula 4.Y ₁(k,τ)=w ₁ ^(H)(k)×(k,τ)  [Mathematical Formula 4]

Herein, H denotes the Hermitian transpose and w₁(k) denotes the adaptivevector for generating the real output. Therefore, the real output andthe dummy output can be expressed in a matrix form by MathematicalFormula 5.

$\begin{matrix}{{y\left( {k,\tau} \right)} = {\left\lbrack \frac{w_{1}^{H}(k)}{{- \gamma_{k}}❘I} \right\rbrack{x\left( {k,\tau} \right)}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 5} \right\rbrack\end{matrix}$

Herein, y(k, τ)=[Y₁(k, τ), Y₂(k, τ), . . . , Y_(M)(k, τ)]^(T),γ_(k)=|Γ_(k) ¹, . . . ,Γ_(k) ^(M-1)|^(T), and Γ_(k)=exp{jω _(k) d sinθ_(target) /c}.

For consistency,

${W(k)} = {\left\lbrack {{W_{1}(k)},\cdots\;,{w_{M}(k)}} \right\rbrack^{H} = {\left\lbrack \frac{w_{1}^{H}(k)}{{- \gamma_{k}}❘I} \right\rbrack.}}$

Nullformer parameters for generating the dummy output are fixed toprovide noise estimation. As a result, according to the presentinvention, permutation problem over the frequency bins can be solved.Unlike an IVA method, the estimation of w₁(k) at a frequency binindependent of other frequency bins can provide fast convergence, sothat it is possible to improve performance of target speech signalextraction as pre-processing for the speech recognition system.

Therefore, according to the present invention, by maximizingindependency between the real output and the dummy output at onefrequency bin, it is possible to obtain a desired target speech signalfrom the real output.

With respect to the cost function, by Kullback-Leibler (KL) divergencebetween probability density functions p(Y₁(k, τ), Y₂(k, τ), . . . ,Y_(M)(k, τ)) and q(Y₁(k, τ))p(Y₂(k, τ), . . . , Y_(M)(k, τ)), the termsindependent of w₁(k) are removed, so that the cost function can beexpressed by Mathematical Formula 6 of which is the target speechextraction method based on the independent component analysis (DC ICA).

$\begin{matrix}{{J_{ICA}^{\prime}(W)} = {- {\sum\limits_{k = 1}^{K}\;\left\{ {{\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} + {E\left. \quad\left\lbrack {\log\mspace{14mu}{q\left( {Y_{1}\left( {k,\tau} \right)} \right)}} \right\rbrack \right\}}} \right.}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Herein, [-]_(m) denotes an m-th element of a vector.

In addition, it may be worth considering the method for target speechextraction based on Independent Vector Analysis(DC IVA) in which thecost function using ŷ_(m)(τ)=[Y_(m) (1, τ), Y_(m)(2, τ), . . . ,Y_(m)(K, τ)]^(Γ) in consideration of the dependency between frequencycomponents sets as follows.

$\begin{matrix}{{J_{IVA}^{\prime}(W)} = {{{- \log}{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} - {E\left\lbrack {\log\mspace{14mu}{q\left( {{\hat{y}}_{1}(\tau)} \right)}} \right\rbrack}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 7} \right\rbrack\end{matrix}$

Because the above-mentioned DC ICA and DC IVA methods converge by usingmaximum gradient algorithm, there is a trade-off between the convergencevelocity due to the step size of learning rate and the stability.

On the other hand, the target speech extraction method using auxiliaryfunction sets the auxiliary function Q and is able to minimize fastwithout need for setting suitably the step size of learning rate. Theauxiliary function Q may be set the inequality relation using aninequality

${G_{R}(r)} \leq {{\frac{G_{R}^{\prime}\left( r_{0} \right)}{2r_{0}}r^{2}} + {G_{R}\left( r_{0} \right)} - \frac{r_{0}{G_{R}^{\prime}\left( r_{0} \right)}}{2}}$so that the auxiliary function has always values greater than or same asthat of the objective function.

Accordingly, through the optimization of the auxiliary function, theobjective function which has always a value smaller than or same as thatof the auxiliary function may be optimized along with it.

In the DC ICA method, the relationship between the auxiliary functionand the objective function is as follows.

$\begin{matrix}\begin{matrix}{{J_{ICA}^{\prime} \leq Q} =} & {{E\left\lbrack {\sum\limits_{k = 1}^{K}\;{\frac{G^{\prime}\left( {r_{1}\left( {k,\tau} \right)} \right)}{2{r_{1}\left( {k,\tau} \right)}}{{Y_{1}\left( {k,\tau} \right)}}^{2}}} \right\rbrack} -} \\ & {{\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} + R} \\{=} & {{\frac{1}{2}{w_{1}^{H}(k)}{V_{1}(k)}{w_{1}(k)}} -} \\ & {{\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} + R}\end{matrix} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 8} \right\rbrack\end{matrix}$

Here, R is a constant irrelevant to the estimated vector w₁(k). In theDC ICA method, V_(m)(k) is as follows.

$\begin{matrix}{{V_{m}(k)} = {E\left\lbrack {\frac{G^{\prime}\left( {r_{m}\left( {k,\tau} \right)} \right)}{r_{m}\left( {k,\tau} \right)}{x\left( {k,\tau} \right)}{x^{H}\left( {k,\tau} \right)}} \right\rbrack}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 9} \right\rbrack\end{matrix}$

Here, r_(m)(k, τ)=|Y_(m)(k, τ)|=|w_(m) ^(H)(k)×(k, τ)|, and G′(r_(m)(k,τ)) denotes a differentiation of G(r_(m)(k, τ))=−log q(Y_(m)(k, τ)) byr_(m)(k, τ).

Meanwhile, it can be expressed a probability density function of variousspeech sources in the time-frequency domain by modeling q(Y_(m)(k, τ))as a probability density function following the Generalized Gaussiandistribution expressed by Mathematical Formula 10.

$\begin{matrix}{{p\left( {Y_{m}\left( {k,\tau} \right)} \right)} \propto {\frac{1}{\lambda_{m}\left( {k,\tau} \right)}\exp\left\{ {- \left( \frac{{{Y_{m}\left( {k,\tau} \right)}}^{2}}{\lambda_{m}\left( {k,\tau} \right)} \right)^{\beta}} \right\}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 10} \right\rbrack\end{matrix}$

Herein, λ_(m)(k, τ) and β are variance and shape parameter,respectively, and the type of pdf can be determined according to thevalue of these parameters. For example, if the pdf is a Laplacedistribution with a unit variance, β=½ and λ_(m)(k, τ)=1, and if the pdfis a Gaussian distribution, then β=1.

In addition, in the DC IVA method, the extended probability densityfunction using clique may be used as follows.

$\begin{matrix}{{p\left( {{\hat{y}}_{m}(\tau)} \right)} \propto {\frac{1}{\Pi_{c = 1}^{N_{c}}{\lambda_{m}\left( {c,\tau} \right)}}\exp\left\{ {- {\sum\limits_{c = 1}^{N_{c}}\;\left( \frac{\Sigma_{k \in \Omega_{c}}{{Y_{m}\left( {k,\tau} \right)}}^{2}}{\lambda_{m}\left( {c,\tau} \right)} \right)^{\beta}}} \right\}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 11} \right\rbrack\end{matrix}$

Herein, c, N_(c) and Ω_(c) are a clique index, a quantity of the cliqueand a set of frequency bin included in the corresponding clique,respectively. Also, λ_(m)(c, τ) and β are a variance and a shapeparameter of the c-th clique in the m-th output, respectively. Herein,it can be expressed the probability density function of various kindsaccording to a design of the clique structure and a setting of the shapeparameter and the variance.

Also, in the DC IVA method based on the clique, the relationship betweenthe objective function and the auxiliary function is as follows.

$\begin{matrix}\begin{matrix}{{J_{IVA}^{\prime} \leq Q} =} & {{E\left\lbrack {\sum\limits_{c = 1}^{N_{c}}\;\left( \frac{\Sigma_{k \in \Omega_{c}}{{Y_{1}\left( {k,\tau} \right)}}^{2}}{\lambda_{1}^{\beta}\left( {c,\tau} \right)} \right)^{\beta}} \right\rbrack} -} \\ & {{\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} + R} \\{=} & {{\frac{1}{2}{w_{1}^{H}(k)}{V_{1}(k)}{w_{1}(k)}} -} \\ & {{\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}} + R}\end{matrix} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 12} \right\rbrack\end{matrix}$

Therefore, v_(m)(k) is given as follows.

$\begin{matrix}{{V_{m}(k)} = {E\left\lbrack {\sum\limits_{c = 1}^{N_{c}}\;{\frac{\left( {\Sigma_{k \in \Omega_{c}}{{Y_{m}\left( {k,\tau} \right)}}^{2}} \right)^{\beta - 1}}{\lambda_{m}^{\beta}\left( {c,\tau} \right)}{x\left( {k,\tau} \right)}{x^{H}\left( {k,\tau} \right)}}} \right\rbrack}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 13} \right\rbrack\end{matrix}$

To optimize the auxiliary function Q to w₁(k), w₁(k) satisfying themathematical formula 14 is obtained.

$\begin{matrix}{{\frac{\partial}{\partial{w_{1}^{*}(k)}}Q} = {{{\frac{1}{2}{V_{1}(k)}{w_{1}(k)}} - {\frac{\partial}{\partial{w_{1}^{*}(k)}}\log{{\sum\limits_{m = 1}^{M}\;{\Gamma_{k}^{m - 1}\left\lbrack {w_{1}^{H}(k)} \right\rbrack}_{m}}}}} = 0}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 14} \right\rbrack\end{matrix}$

Therefore, w₁(k) is satisfied the mathematical formula 15.W(k)V ₁(k)w ₁(k)=e ₁  [Mathematical Formula 15]

Herein, e_(m) is a vector of which the m-th component only is 1 and theother components are all 0. Therefore, w₁(k) satisfies the followingmathematical formula 16.w ₁(k)=(W(k)V ₁(k))⁻¹ e ₁  [Mathematical Formula 16]

w₁(k) may be normalized according to the mathematical formulas 17 or 18as follows.

$\begin{matrix}{{w_{1}(k)} = \frac{w_{1}(k)}{\sqrt{{w_{1}^{H}(k)}{V_{1}(k)}{w_{1}(k)}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 17} \right\rbrack \\{{w_{1}(k)} = \frac{w_{1}(k)}{\sqrt{{w_{1}^{H}(k)}{w_{1}(k)}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 18} \right\rbrack\end{matrix}$

In addition, suppose that λ_(m)(k, τ) is a variance of varying accordingto time or frequency, the method for estimating the λ_(m)(k, τ) can beproposed variously.

For example,

${\lambda_{m}\left( {k,\tau} \right)} = {\beta^{\frac{1}{\beta}}{{Y_{m}\left( {k,\tau} \right)}}^{2}}$can be estimated through the likelihood maximization algorithm, or

${\lambda_{m}\left( {k,\tau} \right)} = {{\beta\left( \frac{{{Y_{m}\left( {k,\tau} \right)}}^{2}}{\lambda_{m}\left( {k,\tau} \right)} \right)}^{\beta - 1}{{Y_{m}\left( {k,\tau} \right)}}^{2}}$can be estimated through the recursive estimation method, where λ_(m)(k,τ) in the right side of the equation can be modified to λ_(m)(k, τ−1).

λ_(m)(k, τ)=¼|Y_(m)(k, τ)| in the Laplace distribution with β=½, andλ_(m)(k, τ)=|Y_(m)(k, τ)|² in the Gaussian distribution with β=1.

Also, λ_(m)(k, τ) can be estimated as the equation of

${\lambda_{m}\left( {k,\tau} \right)} = {\frac{\beta^{\frac{1}{\beta}}}{{2N_{a}} + 1}{\sum\limits_{\tau^{\prime} = {\tau - N_{a}}}^{\tau + N_{a}}\;{{Y_{m}\left( {k,\tau^{\prime}} \right)}}^{2}}}$by considering the value of the adjacent time frame and including N_(α)frames existing on the front and the rear of the current time frame τ,respectively, or λ_(m)(k, τ) can be estimated as the equation of

${\lambda_{m}\left( {k,\tau} \right)} = {\frac{\beta}{{2N_{a}} + 1}{\sum\limits_{\tau^{\prime} = {\tau - N_{a}}}^{\tau + N_{a}}{\left( \frac{{{{w_{m}^{H}(k)}{x\left( {k,\tau^{\prime}} \right)}}}^{2}}{\lambda_{m}\left( {k,\tau^{\prime}} \right)} \right)^{\beta - 1}{{{w_{m}^{H}(k)}{x\left( {k,\tau^{\prime}} \right)}}}^{2}}}}$through the recursive estimation method, where λ_(m)(k, τ) in the rightside of the equation can be modified to λ_(m)(k, τ−1).

Similarly, λ_(m)(k, τ) can be estimated by the recursive equation ofλ_(m)(k, τ)=γ|Y_(m)(k, τ)|²+(1−γ)λ_(m)(k, τ−1), where γ denoted asmoothing parameter. Therefore, λ₁(k, τ) and w₁(k) updates repeatedly inturn to be converged.

In addition, due to Y₁(k, τ)×w₁ ^(H)(k)×(k, τ), λ₁(k, τ) can beinitialized by various methods including above mentioned methods throughthe initial value of w₁(k). If the initial value of w₁(k) is set a unitvector of which the m-th element is 1, the initial value of λ₁(k, τ) canbe measured by a power of input signal of an individual microphone. Inanother method, λ₁(k, τ) can be measured approximately by an improvedsignal power through the beamformer using DOA.

Also, In a method based of the clique, suppose that λ_(m)(c, τ) is avariance varying according to time or clique, the method for estimatingλ_(m)(c, τ) can be proposed variously in the same way.

For example,

${\lambda_{m}\left( {c,\tau} \right)} = {\left( \frac{\beta}{N_{c}} \right)^{\frac{1}{\beta}}\left( {\Sigma_{k \in \Omega_{c}}{{Y_{m}\left( {k,\tau} \right)}}^{2}} \right)}$can be estimated through the maximum likelihood method, or

${\lambda_{m}\left( {c,\tau} \right)} = {\frac{1}{N_{c}}\left( \frac{\Sigma_{k \in {\Omega_{c}{{Y_{m}{({k,\tau})}}}^{2}}}}{\lambda_{m}\left( {c,\tau} \right)} \right)^{\beta - 1}\Sigma_{k \in \Omega_{c}}{{Y_{m}\left( {k,\tau} \right)}}^{2}}$can be estimated through the recursive estimation method, where λ_(m)(c,τ) in the right side of the equation can be modified to λ_(m)(c, τ−1).

In Addition, in the target speech extraction method (DC ICA and DC IVA)using auxiliary function, if the auxiliary function is set with thedistortionless constraint which the input and output signals coming fromthe direction of the target speech using prior information are identicalwith each other, the scaling indeterminacy of signal which is estimatedby a parameter updated in conventional method does not generatefundamentally.

Therefore, it can be obtained the signal having a distortion smallerthan the conventional method, without applying the minimum distortionprinciple MDP to resolve the scaling indeterminacy problem in theconventional method.

If the distortionless constraint w₁ ^(H)(k)h(k)=1, and nullformingconstraint w_(m) ^(H)(k)h(k)=0, (m≠1) are added to the auxiliaryfunction Q, the auxiliary function Q is given as follows.

$\begin{matrix}{Q^{\prime} = {{\sum\limits_{m = 1}^{M}\;\left\lbrack {{\frac{1}{2}{w_{m}^{H}(k)}{V_{m}(k)}{w_{m}(k)}} + {\alpha\left( {{{w_{m}^{H}(k)}{h(k)}} - \beta_{m}} \right)}} \right\rbrack} - {\log{{{\det W}(k)}}} + R}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 19} \right\rbrack\end{matrix}$

Here, h(k) is a direction vector [Γ_(k) ⁰, Γ_(k) ¹, . . . , Γ_(k)^(M-1)]^(T) to a target speech, and β_(m) is 1 in m=1, and 0 in m≠1. Tominimize the above extended auxiliary function Q′, the estimationequation of w₁ ^(H)(k) is given as follows.

$\begin{matrix}{{w_{1}(k)} = \frac{\left( {{W(k)}{V_{1}(k)}} \right)^{- 1}e_{1}}{{h^{H}(k)}\left( {{W(k)}{V_{1}(k)}} \right)^{- 1}e_{1}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 20} \right\rbrack\end{matrix}$

Until now, the method which only w₁ ^(H)(k) is estimated repeatedly byusing the above equation and the remaining w_(m) ^(H)(k) is fixed tonullformer was proposed. However, for performance improvement, theremaining w_(m) ^(H)(k), m≠1 may be estimated repeatedly together byusing the mathematical formula 21.

$\begin{matrix}{\mspace{79mu}{{{w_{m}(k)} = {\left( {{W(k)}{V_{m}(k)}} \right)^{- 1}e_{m}}}{{{a \cdot {w_{m}(k)}} = {\frac{w_{m}(k)}{\sqrt{{w_{m}^{H}(k)}{V_{m}(k)}{w_{m}(k)}}}\mspace{14mu}{or}}},\mspace{20mu}{{w_{m}(k)} = \frac{w_{m}(k)}{\sqrt{{w_{m}^{H}(k)}{w_{m}(k)}}}}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 21} \right\rbrack\end{matrix}$

To induce the real time target speech extraction method based on theauxiliary function, if the inverse matrix of V_(m)(k) in time frame τ isdenoted U_(m)(k; τ) and the inverse matrix of W(k) is denoted A(k; τ),U_(m)(k; τ) can be obtained recursively from U_(m)(k; τ−1), as follows.

$\begin{matrix}{{{U_{m}\left( {k;\tau} \right)} = {\frac{1}{\alpha}\left( {{U_{m}\left( {k;{\tau - 1}} \right)} - \frac{\begin{matrix}{{p_{m}\left( {k,\tau} \right)}{U_{m}\left( {k;{\tau - 1}} \right)}} \\{x\left( {k,\tau} \right){x^{H}\left( {k,\tau} \right)}{U_{m}^{H}\left( {k;{\tau - 1}} \right)}}\end{matrix}}{\begin{matrix}{\alpha + {{p_{m}\left( {k,\tau} \right)}{x^{H}\left( {k,\tau} \right)}}} \\{{U_{m}^{H}\left( {k;{\tau - 1}} \right)}{x\left( {k,\tau} \right)}}\end{matrix}}} \right)}}{{A\left( {k;\tau} \right)} = {{A\left( {k;\tau} \right)} - \frac{{A\left( {k;\tau} \right)}e_{m}\Delta\;{w_{m}^{H}\left( {k;\tau} \right)}{A\left( {k;\tau} \right)}}{1 + {\Delta\;{w_{m}^{H}\left( {k;\tau} \right)}{A\left( {k;\tau} \right)}e_{m}}}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 22} \right\rbrack\end{matrix}$

Here, suppose that w_(m)(k) is w_(m)(k; τ) in the time frame τ. Inaddition,

${p_{m}\left( {k,\tau} \right)} = {\left( {1 - \alpha} \right)\frac{G^{\prime}\left( {r_{m}\left( {k,\tau} \right)} \right)}{r_{m}\left( {k,\tau} \right)}}$and Δw_(m)(k; τ)=w_(m)(k; τ)−w_(m)(k; τ−1) using forgetting factor α.Therefore, w₁(k; τ) can be estimated as follows.w ₁(k;τ)=U ₁(k;τ)A(k;τ)e ₁  [Mathematical Formula 23]

As occasion demands, the norming is performed.

In order to resolve scaling indeterminacy of the output signal byapplying a minimal distortion principle (MDP) to the obtained outputY₁(k, τ)=w₁ ^(H)(k; τ)×(k, τ) obtained by using the w₁(k; τ), thediagonal elements of an inverse matrix of a separating matrix needs tobe obtained.

Due to the structural features, the inverse matrix

$\left\lbrack \frac{w_{1}^{H}\left( {k;\tau} \right)}{{- \gamma_{k}}❘I} \right\rbrack^{- 1}$of the above-described matrix can be simply obtained by calculating onlya factor 1/Σ_(m=1) ^(M)Γ_(k) ^(m-1)[w₁ ^(H)(k; τ)]_(m) for the targetoutput and multiplying the factor to the output.

The real time target speech extraction method which the distortionlessconstraint and nullforming constraint are added to the auxiliaryfunction is exchanged to the mathematical formula 24 from themathematical formula 10.

$\begin{matrix}{{w_{1}\left( {k;\tau} \right)} = \frac{{U_{1}\left( {k;\tau} \right)}{A\left( {k;\tau} \right)}e_{1}}{{h^{H}(k)}{U_{1}\left( {k;\tau} \right)}{A\left( {k;\tau} \right)}e_{1}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 24} \right\rbrack\end{matrix}$

If w_(m) ^(H)(k; τ),m≠1 is estimated repeatedly, the followingmathematical formula.

$\begin{matrix}{\mspace{76mu}{{{{{w_{m}\left( {k;\tau} \right)} = {{U_{m}\left( {k;\tau} \right)}{A\left( {k;\tau} \right)}e_{m}}}{w_{m}\left( {k;\tau} \right)}} = {\frac{w_{m}\left( {k;\tau} \right)}{\sqrt{{w_{m}^{H}\left( {k;\tau} \right)}{V_{m}\left( {k;\tau} \right)}{w_{m}\left( {k;\tau} \right)}}}\mspace{14mu}{or}}},\mspace{20mu}{{w_{m}\left( {k;\tau} \right)} = \frac{w_{m}\left( {k;\tau} \right)}{\sqrt{{w_{m}^{H}\left( {k;\tau} \right)}{w_{m}\left( {k;\tau} \right)}}}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 25} \right\rbrack\end{matrix}$

Next, a time domain waveform of the estimated target speech signal canbe reconstructed by Mathematical Formula 26.

$\begin{matrix}{{y(t)} = {\sum\limits_{\tau}{\sum\limits_{k = 1}^{K}\;{{Y\left( {\tau,k} \right)}e^{j\;{\omega_{k}{({t - {\tau\; H}})}}}}}}} & \left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 26} \right\rbrack\end{matrix}$

FIG. 2 is a flowchart illustrating sequentially the procedure ofalgorithm of the target speech extraction method according to thepresent invention.

FIG. 3 is a table illustrating comparison of calculation amountsrequired for calculating values of the first column of one data framebetween a method according to the present invention and a real-time FDICA method of the related art.

In FIG. 3, M denotes the number of input signals as the number ofmicrophones. K denotes frequency resolution as the number of frequencybins. O(M) and O(M³) denotes a calculation amount with respect to amatrix inverse transformation. It can be understood from FIG. 3 that themethod of the related art requires more additional computations than themethod according to the present invention in order to resolve thepermutation problem and to identify the target speech output.

FIG. 4 is a configurational diagram illustrating a simulationenvironment configured in order to compare performance between themethod according to the present invention and methods of the relatedart. Referring to FIG. 4, there is a room having a size of 3 m×4 m wheretwo microphones Mic.1 and Mic.2 and a target speech source T areprovided and three interference speech sources Interference 1,Interference 2, and Interference 3 are provided. FIGS. 5A to 5I aregraphs of results of simulation of the method according to the presentinvention (referred to as ‘DC ICA’), a first method of the related art(referred to as ‘SBSE’), a second method of the related art (referred toas ‘BSSA’, and a third method of the related art (referred to as ‘RTIVA’) while adjusting the number of interference speech sources underthe simulation environment of FIG. 4. FIG. 5A illustrates a case wherethere is one interference speech source Interference 1 and RT₆₀=0.2 s.FIG. 5b illustrates a case where there is one interference speech sourceInterference 1 and RT₆₀=0.4 s. FIG. 5C illustrates a case where there isone interference speech source Interference 1 and RT₆₀=0.6 s. FIG. 5Dillustrates a case where there are two interference speech sourcesInterference 1 and Interference 2 and RT₆₀=0.2 s. FIG. 5E illustrates acase where there are two interference speech sources (Interference 1 andInterference 2 and RT₆₀=0.4 s. FIG. 5F illustrates a case where thereare two interference speech sources (Interference 1 and Interference 2and RT₆₀=0.6 s. FIG. 5G illustrates a case where three are twointerference speech sources Interference 1, Interference 2, andInterference 3 and RT₆₀=0.2 s. FIG. 5H illustrates a case where threeare two interference speech sources Interference 1, Interference 2, andInterference 3 and RT₆₀=0.4 s. FIG. 5I illustrates a case where threeare two interference speech sources Interference 1, Interference 2, andInterference 3 and RT₆₀=0.6 s. In each graph, the horizontal axisdenotes an input SNR (dB), and the vertical axis denotes word accuracy(%).

It can be easily understood from FIGS. 5A to 5I that the accuracy of themethod according to the present invention is higher than those of themethods of the related art.

FIGS. 6A to 6I are graphs of results of simulation the method accordingto the present invention (referred to as ‘DC ICA’), the first method ofthe related art (referred to as ‘ SBSE’), a second method of the relatedart (referred to as ‘BSSA’), and a third method of the related art(referred to as ‘RT IVA’) by using various types of noise samples underthe simulation environment of FIG. 4. FIG. 6A illustrates a case ofsubway noise and R T₆₀=0.2 s. FIG. 6B illustrates a case of subway noiseand R T₆₀=0.4 s. FIG. 6C illustrates a case of subway noise and RT₆₀=0.6 s. FIG. 6D illustrates a case of car noise and R T₆₀=0.2 s. FIG.6E illustrates a case of car noise and R T₆₀=0.4 s. FIG. 6F illustratesa case of car noise and R T₆₀=0.6 s. FIG. 6G illustrates a case ofexhibition hall noise and R T₆₀=0.2 s. FIG. 6H illustrates a case ofexhibition hall noise and R T₆₀=0.4 s. FIG. 6I illustrates a case ofexhibition hall noise and R T₆₀=0.6 s. In each graph, the horizontalaxis denotes an input SNR (dB), and the vertical axis denotes wordaccuracy (%).

It can be easily understood from FIGS. 6A to 6I that the accuracy of themethod according to the present invention is higher than those of themethods of the related art with respect to all kinds of noise.

FIGS. 7A and 7B illustrate a subband clique and a harmonic clique as twotypical clique cases.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those skilled in the art that various changes in form and details maybe made therein without departing from the spirit and scope of thepresent invention as defined by the appended claims.

A target speech signal extraction method according to the presentinvention can be used as a pre-processing method of a speech recognitionsystem.

What is claimed is:
 1. A target speech signal extraction method ofextracting a target speech signal from input signals input to at leasttwo or more microphones for robust speech recognition, by a processor ofa speech recognition apparatus, comprising: (a) receiving information ona direction of arrival of the target speech source with respect to themicrophones; (b) generating a nullformer for removing the target speechsignal from the input signals and estimating noise by using theinformation on the direction of arrival of the target speech source; (c)setting a real output of the target speech source using an adaptivevector w(k) as a first channel and setting a dummy output by thenullformer as a remaining channel; (d) setting a cost function forminimizing dependency between the real output of the target speechsource and the dummy output using the nullformer by performingindependent component analysis (ICA) or independent vector analysis(IVA); (e) setting an auxiliary function to the cost function; and (f)estimating the target speech signal by using the cost function and theauxiliary function, thereby extracting the target speech signal from theinput signals, wherein the auxiliary function is set an inequalityrelation so that the auxiliary function has always values greater thanor same as that of the cost function.
 2. The target speech signalextraction method according to claim 1, wherein the direction of arrivalof the target speech source is a separation angle θ_(target) formedbetween a vertical line in the microphone and the target speech source.3. The target speech signal extraction method according to claim 1,wherein the nullformer is a “delay-subtract nullformer”, and wherein the(b) includes of obtaining a relative ratio of target speech signals byusing the information on the direction of arrival (DOA) of the targetspeech source, multiplying the relative ratio and an input signal of amicrophone and subtracting the multiplied value from input signals of apair of microphones to cancel out the target speech source componentfrom the input signal of a microphone.
 4. The target speech signalextraction method according to claim 1, wherein a probability densityfunction of the cost function is modeling by a generalized Gaussiandistribution.
 5. The target speech signal extraction method according toclaim 4, wherein the generalized gaussian distribution has a varyingvariance with regard to time-frequency or one of time and frequency, andwherein the (e) includes of updating the varying variance λ and theadaptive vector w(k) alternately, and estimating the target speechsignal by using the updated varying variance and the adaptive vector. 6.The target speech signal extraction method according to claim 4, whereinthe generalized gaussian distribution has a constant variance, andwherein the (e) includes of learning the cost function to update theadaptive vector w(k), and estimating the target speech signal by usingthe updated adaptive vector.
 7. The target speech signal extractionmethod according to claim 4, the target speech signal extraction methodfurther comprises (f) applying a minimal distortion principle (MDP)using a target speech element of a diagonal elements in an inversematrix of a separating matrix, to the estimated the target speech signalin the (e).
 8. The target speech signal extraction method according toclaim 1, wherein a time domain waveform y(k) of an estimated targetspeech signal is expressed by the following Mathematical Formula, and${y(t)} = {\sum\limits_{\tau}{\sum\limits_{k = 1}^{K}\;{{Y\left( {\tau,k} \right)}e^{j\;{\omega_{k}{({t - {\tau\; H}})}}}}}}$wherein Y(k, τ)=w(k)×(k, τ), w(k) denotes an adaptive vector forgenerating a real output with respect to the target speech source, and kand τ denote a frequency bin number and a frame number, respectively. 9.A non-transitory computer readable storage media having programinstructions that, when executed by a processor of a speech recognitionapparatus, cause the processor to perform the target speech signalextraction method according to claim 1.